Search for: All records

Ensuring the correctness of scientific software is challenging due to the need to represent and model complex phenomenon in a discrete form. Many dynamic approaches for correctness have been developed for numerical overflow or imprecision, which may manifest as program crashes or hangs. Less effort has been spent on functional correctness, where one of the most widely proposed technique is metamorphic testing. Metamorphic testing often requires deep domain expertise to design meaningful relations. In this vision paper we ask if we can utilize the process of abstraction and refinement, a traditionally formal approach, to guide the development of metamorphic relations. We have built an iterative approach we call Model Assisted Refinements. It starts with domain-agnostic relations and a set of input-output relations created via a dynamic analysis. We then use a model checker to identify missing input/output patterns and potential passing and failing relations. We augment our dynamic analysis, and obtain domain expertise to verify and refine our relations. At the end we have a set of domain-specific metamorphic relations and test cases. We demonstrate our approach on a high-performance chemistry library. Within three refinements we discover several domain specific relations, and increase our behavioral coverage. 

Small uncrewed aerial systems, sUAS, provide an invaluable resource for performing a variety of surveillance, search, and delivery tasks in remote or hostile terrains which may not be accessible by other means. Due to the critical role sUAS play in these situations, it is vital that they are well configured in order to ensure a safe and stable flight. However, it is not uncommon for mistakes to occur in configuration and calibration, leading to failures or incomplete missions. To address this problem, we propose a set of self-adaptive mechanisms and implement them into a self-adaptive framework,CICADA, for Controller Instability-preventing Configuration Aware Drone Adaptation.CICADAdynamically detects unstable drone behavior during flight and adapts to mitigate this threat. We have built a prototype ofCICADAusing a popular open source sUAS flight control software and experimented with a large number of different configurations in simulation. We then performed a case study with physical drones to determine if our framework will work in practice. Experimental results show thatCICADA’sadaptations reduce controller instability and enable the sUAS to recover from up to 33.8% of poor configurations. In cases where we cannot complete the intended mission, invoking alternative adaptations may still help by allowing the vehicle to loiter or land safely in place, avoiding potentially catastrophic crashes. These safety-focused adaptations can mitigate unsafe behavior in 52.9% to 64.7% of dangerous configurations. We further show that rule-based approaches can be leveraged to automatically select an appropriate adaptation strategy based on the severity of instability encountered, with up to a 14.2% improvement over direct adaptation. Finally, we introduce a variation of our primary adaptation strategy designed to allow more cautious adaptation with limited configuration information, which gets within 6.7% of our primary adaptation strategy despite not requiring an optimal knowledge base. 

Small uncrewed aerial systems (sUAS) are growing in their use for commercial, scientific, recreational, and emergency management purposes. A critical part of a successful flight is a correctly tuned controller which manages the physics of the vehicle. If improperly configured, it can lead to flight instability, deviation, or crashes. These types of misconfigurations are often within the valid ranges specified in the documentation; hence, they are hard to identify. Recent research has used fuzzing or explored only a small part of the parameter space, providing little understanding of the configuration landscape itself. In this work we leverage software product line engineering to model a subset of the parameter space of a widely used flight control software, using it to guide a systematic exploration of the controller space. Via simulation, we test over 20,000 configurations from a feature model with 50 features and 8.88 × 1034 products, covering all single parameter value changes and all pairs of changes from their default values. Our results show that only a small number of single configuration changes fail (15%), however almost 40% fail when we evaluate changes to two-parameters at a time. We explore the interactions between parameters in more detail, finding what appear to be many dependencies and interactions between parameters which are not well documented. We then explore a smaller, exhaustive product line model, with eight of the most important features (and 6,561 configurations) and uncover a complex set of interactions; over 48% of all configurations fail. 

Fault localization is essential to software maintenance tasks such as testing and automated program repair. Many fault localization techniques have been developed, the most common of which are spectrum-based. Most techniques have been designed for traditional programming paradigms that map passing and failing test cases to lines or branches of code, hence specialized programming paradigms which utilize different code abstractions may fail to localize well. In this paper, we study fault localization in the context of a class of programs, molecular programs. Recent research has designed automated testing and repair frameworks for these programs but has ignored the importance of fault localization. As we demonstrate, using existing spectrum-based approaches may not provide much information. Instead we propose a novel approach, Traceback, that leverages temporal trace data. In an empirical study on a set of 89 faulty program variants, we demonstrate that Traceback provides between a 32-90\% improvement in localization over reaction-based mapping, a direct translation of spectrum-based localization. We see little difference in parameter tuning of Traceback when all tests, or only code-based (invariant) tests are used, however the best depth and weight parameters vary when using specification based tests, which can be either functional or metamorphic. Overall, invariant-based tests provide the best localization results (either alone or in combination with others), followed by metamorphic and then functional tests. 

Small Unmanned Aerial Systems (sUAS) must meet rigorous safety standards when deployed in high-stress emergency response scenarios; however many reported accidents have involved humans in the loop. In this paper, we, therefore, present the HiFuzz testing framework, which uses fuzz testing to identify system vulnerabilities associated with human interactions. HiFuzz includes three distinct levels that progress from a low-cost, limited-fidelity, large-scale, no-hazard environment, using fully simulated Proxy Human Agents, via an intermediate level, where proxy humans are replaced with real humans, to a high-stakes, high-cost, real-world environment. Through applying HiFuzz to an autonomous multi-sUAS system-under-test, we show that each test level serves a unique purpose in revealing vulnerabilities and making the system more robust with respect to human mistakes. While HiFuzz is designed for testing sUAS system, we further show that it is applicable across a broader range of Cyber-Physical Systems.